On a Sparse Shortcut Topology of Artificial Neural Networks

In established network architectures, shortcut connections are often used to take the outputs of earlier layers as additional inputs later layers. Despite extraordinary effectiveness shortcuts, there remain open questions on mechanism and characteristics. For example, why shortcuts powerful? Why do generalize well? this article, we investigate expressivity generalizability a novel sparse topology. First, demonstrate that topology can empower one-neuron-wide deep approximate any univariate continuous function. Then, present width-bounded universal approximator in contrast depth-bounded approximators extend approximation result family equally competent networks. Furthermore, with generalization bound theory, show proposed enjoys excellent generalizability. Finally, corroborate our theoretical analyses by comparing popular including ResNet DenseNet, well-known benchmarks perform saliency map analysis interpret Our work helps understand role suggests further opportunities innovate neural architectures.